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ABSTRACT 

Requirements  for  test  and  evaluation  in  the  weapons  system 
acquisition  process  have  become  increasingly  formalized  and 
management  of  the  test  and  evaluation  activities  has  become  more 
centralized,  particularly  with  the  formation  of  the  Major  Range 
and  Test  Facility  Base  and  establishment  of  the  Uniform  Funding 
Policy.   Management  of  the  Navy  portion  of  the  Major  Range  and 
Test  Facility  Base  has  been  delegated  to  the  Naval  Air  Systems 
Command  Test  and  Evaluation  Group;  however,  most  of  the  use  of 
these  facilities  comes  from  the  Systems  and  Engineering  Group. 
Some  of  the  problems  resulting  from  this  divided  responsibility 
are  discussed  here  and  recommendations  for  improvements  are 
Dresented. 
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I.   INTRODUCTION 

A.   STATEMENT  OF  THE  PROBLEM 

In  its  role  of  Developing  Agency  for  Naval  aircraft  weapons 
systems,  including  airframes,  avionics  equipment,  the  Naval  Air 
Systems  Command  (NAVAIR)  must  maintain  the  capability  to  conduct 
test  and  evaluation  (T&E)  of  the  equipment  it  is  developing. 
This  has  been  accomplished  through  the  establishment  and  develop- 
ment of  a  group  of  T&E  field  activities  with  differing  responsi- 
bilities for  the  various  areas  of  interest  for  which  NAVAIR 
requires  testing.   The  role  of  the  T&E  field  activities  is  to 
provide  the  various  groups  at  NAVAIR  Headquarters  with  the 
testing  services  necessary  to  ensure  that  the  various  developmental 
programs  are  progressing  satisfactorily  and  that  they  have  a 
reasonable  probability  of  success.   Also,  Information  must  be 
provided  to  allow  for  correction  of  deficiencies  as  early  as 
possible,  particularly  in  advance  of  expensive  production 
decisions.   Testing  after  a  production  decision  provides  a  form 
of  quality  assurance  and  helps  to  identify  areas  requiring 
further  correction. 

Managing  the  operation  of  the  field  activities,  internally 
and  at  the  Headquarters  level,  in  the  current  environment  of 
weapon  system  acquisition  is  complex  and  difficult.   Several 
issues  have  developed  which  were  discussed  in  a  meeting  of  the 
NAVAIR  Civilian  Management  Board  and  the  Middle  Management 
Council  at  Charlottesville,  Virginia,  in  November,  1977.   Some 
of  these  issues  are  listed  below. 
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1.  There  is  a  lack  of  stated  definitive  "corporate" 
(NAVAIRSYSCOM)  strategy  related  to  roles  of  the  field  activities 
in  the  overall  Naval  Aviation  Program.   The  absence  of  declared 
top  level  strategy  makes  it  difficult  to  develop  a  unified 
corporate  plan  for  field  activity  management  and  has  resulted 

in  a  lack  of  clarity  and  definition  In  roles.  £Q 

2.  There  has  been  a  succession  of  piecemeal  efforts  at 
dealing  with  many  problems  relating  to  field  activity  management 
and  utilization  (e.g.,  decentralization,  distribution  of  civilian 
personnel,  workloading,  etc.)  which  result  in  operating  In  a 
reactive,  non-integrated,  and  non-systematic  mode.  {TJ 

3.  There  are  several  Inconsistent  pressures  effecting  the 
effective  and  efficient  management  of  field  activities  in  the 
area  of  contracting  for  services  (e.g.,  OME  Circular  A-76,  OMB 
Circular  A-109>  Congressional  language  en  FY-1978  contracting 
policies,  MAT-08  policies  on  the  in-house  prosecution  of  R&D).  £Q 

4.  Issues  relating  to  the  modernity  of  the  facilities  and 
equipment  of  the  field  activities  have  not  been  adequately 
treated;  maintenance  and  modernization  of  the  physical  plants 
and  buildings  are  not  treated  in  a  pro-active  mode.  £Q 

5.  Uncertain  institutional  funding  existed  due  to  shortages, 
priority  changes  and  unanticipated  costs.  [Y] 

6.  Unproductive  redundancy  appeared  to  exist.   Fostering 
this  in  part,  was  an  inadequate  communication  process  between 
the  Naval  Air  Systems  Command  (NAVAIR)  Headquarters  and  affected 
field  activities.  £Q 

7.  Different  goals  and  values  existed  as  a  result  of  many 
varying  perceptions  and  Influences  as  well  as  too  many  field 
activity  bosses  (structural,  functional).  QTJ 

8.  Deviations  from  assigned  functional  roles  occurred  due 
in  part  to  the  ready  availability  of  resources  outside  assigned 
mission/function  areas.   These  resources  and  the  related  tasks 
were  viewed  as  tending  to  change  the  basic  reason  for  which  the 
activity  existed  and  for  which  it  was  created.  \jQ 

9.  NAVAIR  does  not  have  adequate  knowledge  and  control  cf 
workload  (functions  and  tasks)  performed  by  field  activities. 
For  example: 

a.  Headquarters  ha3  limited  knowledge  of  the  total  work 
being  performed  by  our  field  activities. 

b.  The  capabilities  cf  our  field  activities  are  net 
adequately  defined. 


c.  The  Navy  Lab  Program  causes  problems  of  prioriti- 
zation and  responsiveness  due  to  the  separation  of  administrative 
and  technical  management. 

d.  Field  activities  have  no  clear  and  disciplined 
statement  of  roles,  missions,  responsibilities,  and  actual 
functions . 

e.  It  is  difficult  to  tie  responsibility /authority/ 
accountability  to  field  activities  for  programs. 

f.  Activities  are  performing  functions  for  organizations 
other  than  NAVAIR,  which  interferes  with  our  priorities. 

g.  Classification  of  activities  as  currently  designated 
is  confusing,  and  in  some  cases  inaccurate.   Example:   PMTC  is 
not  really  just  a  T&E  activity.  QT] 

10.  Functions  and  facilities  at  some  field  activities 
(particularly  RDT&E  activities)  duplicate  and  overlap  each  other. 
Because  of  duplication  of  capabilities,  our  field  activities 
compete  for  the  same  kind  of  work.   For  example  Tactical  software 
systems  are  spread  among  several  activities  causing  duplication 
and  waste.   To  further  compound  the  problem  the  Navy  Lab  Program 
tends  to  duplicate  SYSCOM  and  other  field  efforts.  £-0 

11.  Management  policies,  systems  and  procedures  are  inadeauate 

a.  The  separation  of  administrative  and  technical 
management  in  different  groups  within  NAVAIR  causes  problems  of 
prioritization  and  responsiveness. 

b.  There  is  inadequate  coordination  within  HO  of  field 
workload,  funding  and  manpower  resources  and  tasks. 

c.  There  is  a  critical  imbalance  between  resources  and 
tasks. 

d.  We  have  no  system  to  adjust  ceilings  at  field 
activities  or  between  field  activities  and  Headauarters . 

e.  We  have  no  system  to  balance  facility  requirements 
and  workload  at  field  activities. 

f.  NAVAIR  has  an  Inadequate  workload  control  system 
and  no  means  to  discipline  that  system. 

g.  NAVAIR  field  management  policies  are  unclear. 

h.   Headquarters  provides  field  activities  with  inade- 
quate guidance. 

i.   Our  employees  have  set  up  informal,  inappropriate 
networks  to  allocate  tasks  to  field  activities.  £l] 
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Some  of  the  issues  above  are  overlapping;  however,  it  should 
be  clear  that  a  myriad  of  areas  exist  where  change  is  reauired. 
Some  of  these  are  spoken  of  in  the  past  tense  and  policies  have 
changed  since  November  1977 >    which  would  affect  the  management 
of  the  field  activities,  but  most  of  the  issues  still  exist. 
In  addition  to  the  Issues  cited  by  the  Civilian  Management  Board 
and  the  Middle  Management  Council  there  are  concerns  relative  to 
the  roles  of  the  T&E  field  activities  and  their  relationships 
with  the  Operational  Test  and  Evaluation  Force  (OPTEVFOR), 
problems  associated  with  the  designation  of  most  of  the  T&E 
field  activities  as  components  of  the  Major  Range  and  Test 
Facilities  Base,  and  problems  in  the  application  of  the  Uniform 
Funding  Policy  to  T&E  field  activity  operation. 

B.  OBJECTIVES 

The  objectives  of  this  thesis  are  to  identify  areas  in  the 
management  of  the  NAVAIR  T&E  field  activities  where  improvements 
are  needed  and  to  recommend  changes  to  improve  the  system.   In 
the  discussion  that  follows,  these  areas  of  concern  will  be 
treated  separately  followed  by  recommendations  for  improvement 
in  each  area.   Recommendations  are  briefly  restated  in  the 
Summary  of  Recommendations.   For  the  reader  familiar  with  the 
areas  discussed,  the  recommendations  are  identified  separately 
so  that  the  detailed  discussion  may  be  avoided  if  desired. 

C .  METHODS 

To  accomplish  the  objectives  stated  above,  it  is  first 
necessary  to  examine  the  environment  in  which  NAVAIR  and  its 
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T&E  field  activities  must  operate.   Policies  and  procedures 
that  have  been  established  for  the  management  of  the  T&E  field 
activities  must  be  discussed.   Finally,  the  devices  which  have 
been  developed  to  aid  in  the  management  must  be  analyzed. 

To  determine  the  environment  in  which  NAVAIR  operates, 
specifically  with  regard  to  T&E  and  field  activity  operations, 
a  study  of  various  publications  was  conducted.   This  was  followed 
by  an  examination  of  the  instructions  and  directives  of  NAVAIR, 
the  Naval  Material  Command,  and  DoD  which  establish  the  policies 
and  procedures  for  NAVAIR  management  of  its  T&E  field  activities. 
From  this,  a  list  of  questions  was  generated  to  determine  how 
closely  actual  operations  adhered  to  the  established  policies 
and  procedures.   Responses  to  these  questions  were  requested  from 
personnel  at  NAVAIR  Headquarters  and  the  T&E  field  activities. 
It  Is  important  to  note  that  this  was  not  a  survey  because  the 
questions  did  not  lend  themselves  to  simple  multiple  choice 
answers  and  only  the  few  people  closely  related  to  the  problem 
were  asked  to  respond.   Also,  the  questions  were  not  uniformly 
applicable  to  all  the  respondents.   The  results  of  these  efforts 
were  analyzed  and  interpreted  to  allow  the  objectives  of  the 
thesis  to  be  satisfied. 

D.   BACKGROUND 

Although  problems  still  exist,  defense  systems  acquisition 
has  improved  greatly  in  the  1970 ' s.   Prior  to  this  time  emphasis 
was  on  total  package  procurement;  however,  about  1970  it  was 
realized,  particularly  by  the  Blue  Ribbon  Defense  Panel,  that 
this  concept  was  not  working.   This  was  largely  due  to  over 
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optimistic  cost  and  performance  estimates  and  a  lack  of  compe- 
tition throughout  the  acquisition  process.   Competition  existed 
only  at  the  paper  study  stage  but  design  and  price  competition 
were  nearly  non-existent.   As  a  result  of  the  recommendations  of 
the  Blue  Ribbon  Defense  Panel  and  other  groups,  policies  began 
to  be  developed  which  stressed  demonstrated  performance  as  a 
requirement  for  continued  development.   This  was  further 
emphasized  in  1976  when  Office  of  Management  and  Budget  Circular 
A-109  stressed  that  "each  agency  acquiring  major  systems  should 
provide  strong  checks  and  balances  by  ensuring  adequate  system 
test  and  evaluation."  [2J 

The  policies  of  A-109  are  implemented  in  the  Department  of 
Defense  (DOD)  by  DOD  Directive  5000.1  supported  by  5000.2  and 
5000.3,  which  is  devoted  to  test  and  evaluation.   DODD  5000.1 
states : 

Programs  should  be  structured  and  resources  allocated 
to  ensure  that  the  successful  demonstration  of. program 
objectives  is  the  pacing  activity.  RJ 

Since  demonstration  of  program  objectives  will  require  testing 

it  is  obvious  that  test  and  evaluation  in  the  allocation  of 

defense  resources  has  increased  greatly  in  importance.   Further 

evidence  of  this  is  the  participation  in  all  Defense  Systems 

Acquisition  Review  Councils  (DSARC)  of  the  Deputy  Undersecretary 

of  Defense  for  Research  and  Engineering  (Test  and  Evaluation) 

whose  responsibility  It  is  to  report  to  the  DSARC  and  to  the 

Secretary  of  Defense  on  test  planning  and  results. 

The  increased  importance  of  test  and  evaluation  has  not  been 

without  its  problems.   In  its  report  the  Blue  Ribbon  Defense 
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Panel  recommended  that  each  service  establish  an  operational 
test  and  evaluation  organization  independent  of  both  the  system 
developers  and  the  users.   This  has  resulted  in  the  establishment 
of  the  Operational  Test  and  Evaluation  Agency  (OTEA)  by  the  Army 
and  of  the  Air  Force  Test  and  Evaluation  Center  (AFTEC)  by  the 
Air  Force.   The  Navy  already  had  its  Operational  Test  and  Evalua- 
tion Force  (OPTEVFOR).   All  report  directly  to  the  office  of 
chief  of  staff  of  their  services,  assuring   them  of  independence 
from  the  developers  and  the  users. 

Over  the  past  10  years  changes  have  been  made  in  the  acquisi- 
tion process  which  reduce  concurrency  and  program  risk.   The 
price  has  been  additional  management  reviews,  an  increase  in 
required  test  and  evaluation,  more  use  of  competitive  prototyping; 
all  of  which  greatly  increase  acquisition  cycle  time  and  cost. 
Gansler  has  cited  [VJ  the  effect  in  that  older  systems  such  as 
the  NIKE  AJAX  and  the  HAWK  were  fielded  in  6  years  and  5  years 
respectively;  whereas,  newer  systems  such  as  PATRIOT  and  AEGIS 
will  take  19  and  18  years.   This  increase  in  acquisition  cycle 
time,  and  the  ensuing  efforts  to  reduce  it,  will  result  in 
continual  pressure  to  minimize  the  time  spent  on  T&E  and  will 
directly  impact  the  relationship  between  NAVAIR  and  its  T&E 
field  activities. 
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II.   THE  ENVIRONMENT  OF 
NAVAIR  AND  ITS  T&E  FIELD  ACTIVITIES 

A.   DEVELOPMENTAL  VERSUS  OPERATIONAL  TEST  AND  EVALUATION 

With  the  formation  of  the  operational  test  organizations, 
test  and  evaluation  is  now  formally  divided  into  two  distinct 
areas,  Developmental  Test  and  Evaluation  (DT&E)  and  Operational 
Test  and  Evaluation  (OT&E).   DT&E  is  that  which  is  conducted  by 
or  for  the  Developing  Agency.   Its  first  purpose  is  as  an 
integral  part  of  the  iterative  development  process.   Testing  is 
conducted  to  evaluate  equipment  design  candidates  and  the  results 
are  used  to  modify  the  design.   This  is  followed  by  further 
testing  and  modification.   This  process  continues  until  the 
initial  paper  design  has  been  converted  to  fully  developed 
hardware.   The  second  important  function  of  DT&E  is  to  provide 
information  on  the  progress  of  the  system  development.   Performance 
is  measured  against  a  set  of  time-phased  cost  and  performance  goals 
and  objectives  to  determine  if  the  system  costs  and  performance 
are  at  the  levels  they  should  be  at  a  given  point  in  the  program. 
The  results  can  be  used  as  the  basis  for  continuation  or  adjustment 
to  the  development  program  as  necessary.   A  third  function  of 
DT&E  is  to  help  assure  the  project  manager  that  he  has  a  reasonable 
expectation  of  succeeding  in  the  OT&E  which  follows. 

Early  OT&E  also  aids  in  the  design  process.   To  the  extent 
that  it  is  conducted  on  developmental  hardware  it  helps  to 
identify  operational  deficiencies  to  allow  for  correction  during 
the  development  process.   It  also  allows  the  comparison  of 
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system  performance  with  goals  and  objectives  to  determine  program 
status.   The  difference  between  OT&E  and  DT&E  lies  in  the  orien- 
tation towards  operational  as  opposed  to  technical  factors  and 
the  emphasis  on  predicting  future  performance  rather  than  evalu- 
ating current  performance.   The  primary  purpose  of  T&E  then,  is 
to  provide  answers  to  the  questions,  "What  is  the  program  status?" 
and  "What  is  the  likely  outcome  of  the  program?"  JjQ 

An  important  program  consideration  is  the  amount  of  scarce 
resources  to  be  allocated  to  the  T&E  process.   Since  both  time 
and  money  are  consumed  by  the  T&E  process  without  visibly  advancing 
the  program,  the  temptation  is  to  minimize  T&E.   The  danger  lies 
in  fielding  a  system  which  is  not  technically  or  operationally 
suitable . 

The  problem  is  compounded  by  the  fact  that  it  is  often 
difficult  to  categorize  tests  as  operational  or  developmental. 
Policies  have  been  established  which  discourage  joint  testing 
unless  it  is  dictated  overwhelmingly  by  the  costs  of  conducting 
separate  developmental  and  operational  tests.   The  tendency, 
therefore,  is  towards  duplication  of  testing.   Since  the  opera- 
tional test  agencies  have  no  obligation  to  minimize  overall 
program  costs  or  maintain  the  program  schedule,  there  might  be 
a  tendency  to  over  test  and  to  be  over  critical  in  their  evalu- 
ation of  the  results  to  aggrandize  their  own  position  in  the 
acquisition  process.   In  the  case  of  developmental  testing  the 
situation  is  similar  but  slightly  better.   The  developmental 
testers  work  for  the  program  manager  and  are  dependent  on  him 
for  their  funding;  therefore,  he  has  discretion  in  the  amount 
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funds  and  time  allocated  to  developmental  testing.   The  dilemma 
of  the  program  manager  is  to  decide  how  much  time  and  money  to 
spend  on  developmental  testing.   The  temptation  is  to  over- 
emphasize operational  testing  since  he  has  little  discretion  in 
this  area,  thereby  minimizing  his  testing  costs.   The  danger  In 
that  approach  is  that  a  system  that  has  not  had  the  benefit  of 
sufficient  test  and  modification  of  the  iterative  developmental 
testing  process  may  be  killed  by  the  sometimes  overzealous 
operational  testers. 

The  culmination  of  OT&E  is  the  OPEVAL  which  must  be  completed 
prior  to  approval  for  full  scale  production.   This  is  preceded 
by  a  Certification  of  Readiness  for  OPEVAL  which  must  be  provided 
to  ensure  that  the  system  to  be  tested  is  sufficiently  represen- 
tative of  the  system  that  will  ultimately  be  deployed.   Procedures 
for  the  Certification  are  provided  by  NAVAIRINST  3960.2A  and 
include  a  number  of  actions  required  by  the  various  NAVAIR  com- 
ponents under  the  overall  coordination  of  AIH-05.   In  theory  the 
process  is  relatively  straightforward.   In  practice  it  has  been 
a  significant  problem  area.   Problems  have  arisen  in  determining 
that  the  systems  proposed  for  OPEVAL  are  representative  of 
production  systems.   Because  the  DT&E  Inevitably  points  up 
deficiencies  which  must  be  corrected  in  production,  major 
differences  exist  between  the  full  scale  development  models  and 
the  production  items.   Correction  of  these  deficiencies  and 
configuration  differences  prior  to  OPEVAL  is  often  time  consuming 
and  costly.   A  further  requirement  for  OPEVAL  is  that  logistic 
support,  manning,  and  training  for  the  operation  of  the  system 
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during  OPEVAL  be  representative  of  fleet  deployment.   This  is 
often  not  possible  without  considerable  delay  because  development 
of  support  equipment,  training  of  personnel,  and  availability  of 
spares  in  the  supply  system  lag  significantly  behind  system 
development.   Readiness  for  OPEVAL,  therefore,  becomes  a  process 
of  negotiation  rather  than  a  check-off  of  tne  elements  as  intended 
This  is  a  delicate  process  for  the  program  manager,  since  he  must 
balance  program  schedule  and  cost  constraints  against  the  risks 
of  failing  to  satisfy  OPEVAL  requirements  for  lack  of  some  of 
the  items  mentioned  above. 

One  of  the  primary  objectives  of  OT&E,  and  particularly  the 
OPEVAL,  is  that  it  be  conducted  under  conditions  representative 
of  those  to  be  expected  in  field  operations.   The  reasons  for 
this  are  obvious;  however,  in  practice  it  is  often  difficult  to 
achieve  for  the  following  reasons: 

1.  Inadequate  threat  environment. 

The  operating  conditions  include  the  enemy  weapons 
systems  and  their  operational  doctrine  against  which  the  system 
under  test  must  operate.   Simulations  of  these  systems  do  exist, 
adequate  for  developmental  testing;  however,  large  scale  networks 
adequate  to  simulate  the  complete  operational  environment  are 
extremely  expensive  and  do  not  exist. 

2.  Insufficient  numbers  of  test  systems. 

For  the  same  reasons  it  is  desirable  to  operate  against 
a  large  scale  threat  environment,  it  is  also  desirable  to  operate 
the  system  under  test  in  conjunction  with  all  the  other  systems 
it  is  expected  to  live  with  and  in  the  manner  it  is  expected  to 
be  operated.   This  is  also  extremely  expensive  and  not  generally 
done  except  for  individual  compa tability  tests. 

3.  Inadequate  training  of  personnel. 

The  objective  is  to  conduct  the  test  utilizing  personnel 
that  will  operate  the  system  in  the  field.   Since  schools  have 
usually  not  been  established  prior  to  a  production  decision, 
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adequately  trained  personnel  do  not  exist.  The  systems  must  be 
maintained  by  persons  of  higher  skill  level  than  the  operators, 
usually  contractor  personnel. 

4.  System  immaturity. 

In  today's  environment  emphasis  is  placed  on  reliability, 
The  systems  tested;  however,  have  been  built  by  different  tech- 
niques than  will  be  used  in  full  scale  production.   They  will 
suffer  from  "infant  mortality"  problems  which  the  mature  system 
will  not  have.   Also,  with  the  limited  number  of  systems  and 
operating  hours  available,  a  statistically  valid  estimate  of 
reliability  usually  cannot  be  determined. 

5.  Lack  of  production  configured  support  equipment  and 
publications . 

This  is  related  to  the  maintenance  personnel  problem. 
The  support  equipment  designs  and  the  publications  cannot  be 
determined  until  the  production  configuration  of  the  prime 
equipment  is  fairly  well  defined.   Therefore,  they  are  usually 
not  ready  for  testing  in  time  and  modified  general  purpose  test 
equipment  must  be  used. 

For  obvious  reasons  it  is  desirable  that  objectivity  and 
independence  be  maximized  in  the  test  program.   This  is  particu- 
larly true  of  the  OT&E.   Since  the  DT&E  is  conducted  under  the 
direction  of  the  Developing  Agency,  biases  are  almost  certain  to 
be  present.   Evaluation  of  the  results  is  often  difficult  under 
these  conditions.   In  the  OT&E  the  danger  exists  that  "nice  to 
have"  features  will  be  interpreted  as  necessary  since  OT&E  is 
generally  conducted  without  regard  to  system  cost.   Another  less 
likely  possibility  is  to  minimize  deficiencies  in  seeking  early 
deployment.   Factors  which  alleviate  this  problem  are  the  checks 
and  balances  between  the  DT&E  and  OT&E. 

Previously,  emphasis  on  OT&E  was  at  program  stages  just 
prior  to  the  production  decision  because  of  the  large  funding 
commitment  at  that  point.   Lately  it  has  been  realized  that 
emphasis  on  earlier  OT&E  is  needed  for  four  reasons. 
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1.  In  the  early  stages  of  system  concept  formulation  It  Is 
necessary  to  know  that  the  concepts  are  tactically  sound. 

2.  To  have  any  impact  on  system  development  OT&E  must  be 
conducted  earlier  than  just  prior  to  the  production  decision 
since  by  then  most  of  the  research  and  development  resources 
have  been  expended. 

3.  Cost  and  schedule  impact  can  be  minimized  by  early 
detection  and  correction  of  operational  deficiencies. 

4.  Commitment  of  funds  for  limited  production  prior  to  the 
major  production  decision  requires  earlier  availability  of  test 
results.  QQ 

The  role  of  OPTEVFOR  is  defined  in  Reference  6.   It  directs 

OPTEVFOR  involvement  in  all  phases  of  the  acquisition  cycle  and 

provides  for  close  coordination  between  OPTEVFOR  and  the  developing 

agency.   Specifically,  during  the  program  validation  phase  OPTEVFOR 

will: 

Establish  liaison  with  the  developing  agency  for 
providing  operational  planning  assistance  in  determining 
the  testing  required  for  the  current  project,  anticipated 
future  testing  requirements,  and  defining  critical  opera- 
tional issues.  £6j 

Participate  in  test  planning  and  independently  observe 
selected  tests  and  demonstrations  conducted  by  the  develop- 
ing agency,  and  examine  test  results.   submit  an  assess- 
ment of  this  initial  test  and  evaluation  to  the  Chief  of 
Naval  Operations,  and  the  developing  agency,  including 
comments  and  recommendations  concerning  future  operational 
suitability  of  the  system,  progress  to  date,  and  operational 
Issues  requiring  further  examination.  \o~] 

During  the  full  scale  engineering  development  phase  OPTEVFOR 

will: 

Independently  observe  selected  developmental  tests 
and  view  the  data  from  an  operational  viewpoint,  to 
verify  readiness  for  operational  testing.  jjSJ 

From  this  it  can  be  seen  that  much  of  the  work  of  the  TScE 

field  activities  will  be  evaluated  by  OPTEVFOR;  therefore,  it  is 

necessary  to  be  continually  aware  of  their  opinions  and  priorities. 
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Test  programs  may  have  to  be  adjusted  to  help  satisfy  OPTEVFOR 
requirements.   Since  ultimately  responsibility  for  responding  to 
criticisms  of  the  test  programs  will  fall  to  the  program  manager 
and  his  team,  they  should  closely  monitor  the  planning,  conduce, 
and  results  of  the  tests  conducted  by  the  field  activities.   This 
will  be  discussed  further  in  a  later  section. 

B.   THE  MAJOR  RANGE  AND  TEST  FACILITY  BASE 

In  1970  the  Blue  Ribbon  Defense  Panel  recognized  the  need 
for  better  test  and  evaluation  management  and  recommended  a  major 
DOD-wide  study  of  test  and  evaluation  facilities.   The  study  was 
completed  in  August  1971  and  resulted  in  the  formation  of  the 
Major  Range  and  Test  Facility  Base  (MRTFB),  which  consisted  of 
26  T&E  activities  of  the  Army,  Navy,  and  Air  Force  with  a  combined 
annual  budget  in  excess  of  $1  billion.  Pf]  The  activities 
currently  comprising  the  MRTFB  are  listed  in  Appendix  A. 

Policies  for  the  use,  management  and  operation  of  MRTFB 

activities  are  established  in  Reference  8.   The  mission  of  the 

MRTFB  is  to: 

Provide  a  broad  range  and  test  support  base  to  all  DOD 
Components  responsible  for  development,  test,  evaluation 
and  operation,  as  applicable,  of  material  and  weapon  systems, 
and  to  other  Federal  Government  Agencies  having  need  for 
that  support.  T8J 


Overall  responsibility  within  DOD  is  assigned  to  the  Deputy 
Undersecretary  of  Defense  for  Research  and  Engineering,  Test 
and  Evaluation  (DD(T&E)).   Authorities  and  responsibilities  for 
DD(T&E),  Secretaries  of  the  military  departments,  Facility 
commanders,  and  Facility  users  are  established.   Reference  9 
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assigns  responsibility  for  management  of  Navy  portions  of  the 
MRTFB  to  the  NAVAIR  Assistant  Commander  for  Test  and  Evaluation 
(AIR-06). 

C.   THE  UNIFORM  FUNDING  POLICY 

Prior  to  fiscal  year  1975  no  single  policy  existed  for 
assessing  the  costs  of  operation  of  test  facilities.   Some 
activities  were  funded  institutionally.   That  is,  money  for 
their  operations  was  provided  from  a  separate  appropriation 
independent  of  any  program  utilizing  the  test  facilities.   A 
second  method,  industrial  funding,  required  that  facility  users 
pay  all  the  costs,  including  overhead,  of  operating  a  facility. 
A  third  policy  required  that  users  pay  only  the  direct  costs 
associated  with  their  tests,  with  the  remainder  of  costs  to  be 
paid  from  institutional  funds.   There  were  also  combinations  of 
these  methods  which  sometimes  varied  depending  on  who  was  the 
user . 

The  study  completed  in  August  1971*  noted  that  because  of 
the  differences  in  funding  policy,  test  site  selection  was  often 
based  on  cost  to  the  user  instead  of  test  site  capabilities.   As 
a  result,  in  January  1973*  the  Deputy  Secretary  of  Defense 
directed  the  establishment  of  a  Uniform  Funding  Policy  at  18  of 
the  26  MRTFB  activities,  with  users  paying  direct  costs  and 
indirect  costs  being  paid  from  institutional  funds.   The  policy 
was  incorporated  into  DODD  3200.11  and  went  into  effect  in 
fiscal  year  1975.   The  Navy  activities  to  which  the  Uniform 
Funding  Policy  applies  are  the  Naval  Air  Propulsion  Center, 
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Naval  Air  Test  Center,  Naval  Weapons  Center,  Pacific  Missile 
Test  Center,  and  the  Atlantic  Undersea  Test  and  Evaluation 
Center. 

DODD  3200.11  states  that  the  intent  of  the  Uniform  Funding 
Policy  as  follows: 

1.  Provide  for  interservice  compa tability,  efficiency  and 
equity,  without  influencing  technical  decisions  on  testing,  nor 
inhibiting  legitimate  and  valid  testing. 

2.  Reimbursement  for  use  of  the  ranges  and  test  facilities 
will  be  required. 

3.  All  the  costs  of  the  range  or  test  facilities  which  are 
not  reimbursed  by  users  will  be  funded  by  the  management  agency 
of  the  particular  range  or  facility. 

4.  All  DOD  Components  and  other  Federal  Government  agencies 

will  reimburse  the  ranges  and  test  facilities  for  direct 

costs,  excluding  military  labor. 

5.  Direct  Costs.  Those  expenses  which  can  be  immediately 
and  directly  identified  with  a  specific  User  program  (usually 
documented  by  a  job  order).  These  costs  include  direct  labor, 
direct  material,  minor  construction,  special  purpose  equipment 
and  other  like  costs.  They  include  all  such  expenses  that  can 
with  reasonable  effort  be  identified  consistently  and  uniformly 
to  specific  User  programs.   The  direct  cost  for  labor  includes 

a  load  factor  covering  items  such  as  leave  and  contributions  to 
the  retirement  program.  00 

In  concept  the  Uniform  Funding  Policy  is  relatively  simple. 

The  implementation,  however,  has  led  to  problems.   Historically 

it  has  been  difficult  to  predict  the  workload  of  test  activities 

with  any  accuracy,  even  for  the  following  fiscal  year.   Without 

an  accurate  estimate  of  direct  funding  it  is  impossible  to 

accurately  budget  the  institutional  funds  required.   Expenditures 

for  civilian  personnel  are  relatively  fixed  and  pricrity  for  use 

of  institutional  funds  must  be  in  this  area.   The  impact  is  on 

capital  expenditures  for  new  test  capabilities.   An  orderly 

program  is  difficult  to  establish  because  funds  must  be  held  Ir. 
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escrow  against  the  possible  shortfall  of  direct  funding.   This 
will  be  discussed  in  a  later  section. 

Problems  also  result  for  the  users  of  the  test  facilities 
because  of  program  delays.   If  the  user  plans  a  test  for  a  given 
fiscal  year,  allocates  the  money  to  the  test  activity,  and  because 
of  system  problems  the  test  is  delayed  beyond  the  end  of  the 
fiscal  year,  the  program  manager  finds  himself  at  the  end  of 
the  fiscal  year  with  the  allocated  test  funds  he  can  no  longer 
use  and  he  still  must  provide  test  funding  out  of  the  following 
year's  budget. 

Increases  in  administrative  workload  as  a  result  of  the 
implementation  of  the  Uniform  Funding  Policy  have  been  docu- 
mented. £9j  These  increases  were  due  largely  to  the  increased 
budgeting  requirements  and  the  need  for  more  detailed  estimates 
and  reports.   The  increased  workload  has  largely  been  absorbed 
with  existing,  already  limited,  personnel  resources.   The  increased 
workload  was  not  limited  to  comptroller  personnel,  but  also 
affected  technical  personnel  and  detracted  from  their  attention 
to  their  technical  duties. 

D.   THE  BUDGET  PROCESS 
1.   Discussion 

The  process  by  which  NAVAIR  must  prepare  budgets  for 
T&E  field  activity  operations  is  complicated  by  the  fact  that 
they  are  funded  from  multiple  appropriations.   Institutional 
funding  is  provided  from  the  Category  V,  Management  and  Support, 
funds.   User  funding  may  come  from  Research,  Development.  Test 
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and  Evaluation,  Navy  (RDT&EN)  funds  or  from  one  of  the  procurement 

accounts,  depending  on  the  actual  project  being  prosecuted.   The 

problem,  then,  becomes  one  of  balancing  the  amounts  budgeted  from 

each  appropriation  to  avoid  violation  of  Section  3678  of  the  U.S. 

Code  which  states  that  funds  must  only  be  spent  for  the  purpose 

for  which  they  are  appropriated.   Reference  10  states: 

Apportionment  of  funds  Includes  an  amount  for  unfunded 
reimbursements  expected  to  be  earned  during  the  fiscal 
year.   The  allocation  of  these  anticipated  reimbursements 
must  not  exceed  the  expected  year-end  gross  unfunded 

accounts  receivable.  Work  supported  by  reimbursable 

orders  is  to  be  financed  within  the  subhead  under  which 
the  order  is  accepted. 

Budget  estimates  for  Navy  procurement  appropriations  are 
submitted  by  NAVAIR  to  NAVMAT  where  they  are  consolidated  and 
submitted  to  the  Office  of  Budget  and  Reports/Fiscal  Management 
Division  which  is  responsible  for  formulation  of  the  total  Navy 
budget.   Budget  estimates  for  the  RDT&EN  appropriation  are 
submitted  by  NAVAIR,  usually  by  a  different  office  than  for  the 
procurement  appropriations,  to  the  Deputy  Chief  of  Naval  Material 
for  Development  where  they  are  coordinated  and  submitted  to  the 
Office  of  Naval  Research  for  consolidation  and  ultimate  submission 
to  the  Office  of  Budget  and  Reports/Fiscal  Management  Division. 

The  separation  of  budget  responsibilities  makes  consolidation 
of  the  various  appropriations  budgets  into  a  single  budget 
estimate  for  T&E  field  activity  operations  difficult,  if  not 
impossible.   Paragraph  7  of  Reference  11  requires  that  NAVAIR 
users  of  the  T&E  field  activities  submit  written  fiscal  plannl 
information  to  AIR-6103,  the  T&E  Field  Activity  Support  3ranch 
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of  AIR-06.   This  would  help  alleviate  the  problem;  however, 
responses  to  questions  do  not  Indicate  that  the  policy  is  being 
followed. 

Another  problem  in  the  budgeting  process  is  the  time  limita- 
tion on  the  availability  of  funds.   Congress  enacts  RDTScEN 
appropriations  for  two  years  but  Navy  policy  is  "to  program 
RDT&E  effort  on  an  annual  incremental  funding  basis  as  opposed 
to  the  fully  funded  basis  on  which  procurement  programs  are 
developed."  (j.l]  The  Navy  Comptroller  Manual  states: 

Although  the  RDT&EN  appropriation  is  legally  available 
for  obligation  for  2  fiscal  years,  the  objective  is  to 
use  these  funds  during  the  initial  year  of  availability. 
The  budget  should  be  formulated  to  reflect  this  funding 

objective.  However,  if  the  award  of  a  planned  new 

contract,  or  the  issuance  of  a  reimbursable  order,  is 

not  possible  until  the  second  year  of  availability,  

the  preferred  action  is  to  restructure  the  funding  plan 
to  utilize  the  original  funding  in  the  first  year  of 
availability  for  urgent  deficiencies  elsewnere  in  the 
program  and  to  use  the  following  year's  appropriation 
to  fund  new  contracts  or  orders.  Q.lJ 

This  policy  exacerbates  the  problem  mentioned  earlier  in 
reference  to  the  Uniform  Funding  Policy.   Program  managers  need 
not  lose  their  funds  due  to  program  slippages  if  the  money  is 
legally  available  for  another  year. 

2.   Recommenda  tions 

The  policy  requiring  users  of  the  TScE  field  activities 
to  submit  written  fiscal  planning  information  to  AIR-6102,  if 
followed,  would  improve  the  ability  to  prepare  accurate  consoli- 
dated budget  estimates  for  TScZ  field  activity  operations. 
Adherence  to  this  policy  should  be  further  emphasized  3S  it 
would  help  give  NAVAIR  managers  3  better  grasp  of  the  overall 
costs  of  T&E  in  the  acquisition  process. 
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The  policy  which  requires  obligation  of  all  of  the 
RDT&EN  funds  of  a  fiscal  year  appropriation  within  the  first 
year  of  availability  should  be  reexamined.   There  may  be  more 
advantages  than  disadvantages  in  allowing  the  money  to  be 
obligated  throughout  the  full  two  years  it  is  available. 

E.   MAJOR  AND  LES3-THAN-MAJ0R  SYSTEM  ACQUISITIONS 

Policies  and  procedures  for  major  systems  acquisitions  have 
been  well  established  and  documented  at  length.   This  area  has 
received  considerable  attention  because  of  the  large  sums  of 
money  involved.   For  less-than-ma jor  systems  the  3um  involved 
in  a  single  acquisition  is  smaller  and,  therefore,  of  less 
importance.   Nevertheless,  when  all  of  the  less-than-ma jor 
systems  acquisitions  are  considered  in  aggregate  the  amount 
involved  is  not  trivial.   Unfortunately,  for  these  systems  che 
procedures  have  not  been  as  well  established,  particularly  with 
regard  to  establishing  T&E  requirements  and  planning  to  satisfy 
those  requirements. 

One  reason  for  the  problem  is  the  volatility  of  the  acquis  icier, 
process.   DOD  Directive  5000.1,  the  document  that  prescribes 
policies  and  procedures  for  major  system  acquisition,  has  teen 
rewritten  twice  within  the  last  five  years  and  is  currently 
being  rewritten  again.   Below  the  level  of  DODD  5000. 1  there  are 
numerous  instructions  and  directives  which  implement  the  broad 
policy  guidelines.   When  the  major  policy  document  changes  as 
rapidly  as  it  has,  these  subordinate  documents  almost  never 
catch  up.   This  results  in  directives  which  are  often  in  cc     ^c 
with  each  other  and  it  is  often  impossible  to  be  in  compliance 

27 


with  all  the  current  Instructions.   Also,  programs  are  generally 
of  such  duration  that  they  outlast  changes  in  policy.   Thi3  is 
particularly  true  when  it  is  realized  that  many  programs  consist 
of  derivatives  of  existing  programs  and,  therefore,  do  not  fit 
Into  the  current  phases  of  the  acquisition  process. 

Policies  and  procedures  for  test  and  evaluation  established 
by  higher  authority  are  implemented  within  NAVAIR  by  NAVAIRINST 
3960. 2A  which  states: 

The  provisions  of  this  instruction  apply  to  the  following: 

All  acquisition  projects  including  thc3e  classified  as 
Acquisition  Categories  (ACAT)  I,  II,  III,  and  IV  projects. 

Projects  involving  significant  modifications  tc  existing 
systems  and  equipment  whose  Research,  Development,  Test  and 
Evaluation  (RDT&E)  and  Procurement  Appropriation  Funding 
Requirements  and  other  criteria  would  qualify  them  as  ACAT 
I,  II,  III,  or  IV  projects. 

Individual  subsystems,  equipment  or  components  developed 
as  a  part  of  a  major  system  such  as  a   radar  equipment,  a  computer, 
acoustic  processor,  etc.  which  are  susceptible  to  separate  T&E 
and  which  are  required  by  competent  authority  to  be  the  subject 
of  a  separate  Test  and  Evaluation  Master  Plan  (TEMP)  or  Test  and 
Evaluation  Plan  (TEP). 

"Off-the-shelf"  commercial  systems  and  equipments  or  signi- 
ficant modifications  thereof  when  the  program  decision  authority 
approves  this  alternative  in  lieu  of  new  development  to  meet  an 
established  mission  need. 

Individual  subsystems,  equipment  or  components  which  are  not 
developed  as  a  part  of  a  major  weapon  system  and  which  are 
susceptible  to  separate  T&E  and  are  required  by  competent 
authority  to  be  the  subject  of  a  separate  TEMP  to  TEF.  Q2J 

Definitions  of  the  Acquisition  Categories  are  given  in 

Appendix  B.   The  TEMP  and  TEP  will  be  discussed  in  a  later 

section.   From  the  statement  above  and  the  definition  of  the 

Acquisition  Categories  it  can  be  seen  chat  It  is  unlikely  that 

the  provisions  of  a  formal  T&E  program  would  not  apply  to  most 
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acquisitions.   NAVAIRINST  3960. 2A  is  relatively  new  and  it  will 
take  some  time  to  determine  if  the  policies  are  followed  for  all 
acquisition  projects;  however,  if  it  is  applied,  it  will  go  a 
long  way  towards  improving  the  budgeting  of  T&E  field  activity 
operations.   This  is  especially  important  in  today's  environment 
of  conversions  in  lieu  of  procurement  and  service  life  extension 
programs. 

F.   THE  TEST  AND  EVALUATION  MASTER  PLAN 

To  help  make  the  T&E  of  major  3y3tems  an  orderly  process, 
the  program  manager  is  required  by  OPNAVINST  396O.IO  to  submit 
a  Test  and  Evaluation  Master  Plan  (TEMP).   The  TEMP  is  the 
controlling  T&E  planning  document  prepared  by  NAVAIR  In  collabor- 
ation with  NAVMAT,  OPTEVFOR,  and  the  Board  of  Inspection  and 
Survey,  when  applicable,  and  Is  approved  by  OPNAV.   12  It 
serves  as  a  contract  between  the  Developing  Agency  and  the  user 
for  the  T&E  to  be  conducted  throughout  the  program.   Included 
are  information  on  program  goals  and  milestones,  system  parameters, 
performance  and  cost  thresholds,  and  test  resource  requirements. 
The  test  schedule  and  the  level  of  testing  to  be  conducted  are 
keyed  to  the  major  project  milestones.   Separate  sections  are 
provided  for  DT&E  and  OT&E.   The  DT&E  section  is  prepared  by 
NAVAIR  and  the  OT&E  section  is  prepared  by  OPTEVFOR.   Emphasis 
is  placed  on  resolving  issues  prior  to  submittal  to  OPNAV; 
however,  sometimes  this  is  not  possible  and  the  TEMP  must  be 
submitted  with  open  Issues  for  OPNAV  resolution. 
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The  TEP  is  the  controlling  T&E  planning  document  for  ACAT 
IV  projects  and  is  approved  by  the  Deputy  Commander,  NAVAIR. 
It  is  prepared  within  NAVAIR  and  contains  the  same  type  of 
planning  information  contained  in  the  TEMP  for  larger  programs. 

Prior  to  seeking  a  Milestone  I  decision  the  program  manager 
is  required  to  have  the  T&E  plan  prepared.   For  major  programs 
it  is  required  as  an  annex  to  the  Decision  Coordinating  Paper, 
the  document  used  as  the  basis  for  the  Defense  Systems  Acquisition 
Review  Council  recommendation  and  the  Secretary  of  Defense  decislcr. 
The  initial  version  is  to  avoid  specifics  of  the  test  program 
and  concentrate  on  the  broad  test  plan  and  the  critical  tests 
required  for  satisfying  major  project  objectives.   Also,  it  is 
to  identify  the  total  resource  needs  for  the  test  program  in 
terms  of  manpower,  material,  facilities  and  funding.   The  plan 
is  to  be  revised  during  the  life  cycle  to  add  details  as  they 
become  known.   The  revision  prior  to  the  Milestone  II  decision 
should  contain  sufficient  test  program  specifics  to  ensure  that 
the  planned  testing  is  sufficient  to  demonstrate  program 
requirements . 

Within  NAVAIR  Headquarters  AIR-OS,  the  Systems  and  Engineering 
Group,  is  responsible  for  the  technical  content  of  the  T&E  plan 
and  technical  direction  of  the  T&E  program.   AIR-06  is  responsible 
for  integrating  the  various  elements,  coordinating  the  staff 
review  of  documents,  coordinating  with  the  T&E  field  activities 
and  arranging  for  the  required  AIR-06  managed  resources. 

The  use  of  T&E  planning  documents  has  the  potential  to  greatly 
improve  the  forecasting  of  user  requirements  and  funding  for  the 
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T&E  field  activities  and  to  aid  in  overall  forecasting  of  field 
activity  workload  and  requirement  for  institutional  funding. 
The  effectiveness  and  efficiency  of  the  TEMP  process  has  not 
been  optimized,  however,  because  it  has  not  been  sufficiently 
integrated  with  the  other  information  systems  available  to 
provide  a  composite  picture  for  all  projects  and  all  field 
activities. 
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III.   THE  RELATIONSHIP  OF  THE  ELEMENTS 
OF  T&E  FIELD  ACTIVITY  CONTROL 

A.  APPLICABLE  DIRECTIVES  AND  INSTRUCTIONS 

A  list  of  most  of  the  major  directives  and  instructions 
applicable  to  the  structure  and  management  of  NAVAIR  T&E  programs 
and  the  operation  of  the  T&E  field  activities  is  included  in 
Appendix  C. 

B.  DIVISIONS  OF  RESPONSIBILITY 
1.   Discussion 

Policies  and  procedures  for  planning  and  conducting 

NAVAIR  T&E  programs  are  contained  in  NAVAIR  Instruction  3960.2A. 

Overall  project  responsibility  is  delegated  to  the  program 

managers  (PMA)  for  the  individual  programs;  however,,  the  primary 

functional  responsibilities  for  T&E  are  delegated  to  AIR-06  and 

AIR-05.  the  Systems  and  Engineering  Group.   Figures  1  and  2  show 

the  relationship  of  NAVAIR  with  external  organizations  and  the 

relationship  of  the  various  elements  within  NAVAIR.   Gucting 

from  396O. 2A: 

The  MATACQ  Group  (AIR-05 )  is  line  responsible  for  the 
performance  of  the  full  set  of  acquisition  functions 
within  the  matrix  project  management  concept  except  for 
those  specific  program  areas  assigned  to  other  Headquarters 
Groups  (e.g.,  Target  Systems).   Accordingly,  Air-05  is 
responsible  for  coordination  of  the  implementation  of  the 
comprehensive  T&E  programs  and  the  planning,  execution, 
and  technical  evaluation  of  the  DT&E  and  PAT&E  (except  for 
BIS  Acceptance  Trials)  elements  thereof.   The  T&E  Group 
(AIR-Ob)  is  the  designated  single  point  of  contact  for 
NAVAIR  T&E  program  matters  and  as  such  is  line  responsible 
for  the  Interpretation  and  implementation  of  all  T&E 
policies  and  procedures.   AlR-Ob  functions  to  develop 
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and   promulgate   appropriate   NAVAIR  policies   and    procedures 
to   provide    T&E   guidance   and   assistance    to   all    organizational 
units    within   NAVAIR.      AIR-06   also   provides    NAVAIR  and    Navy- 
wide   management   of  specifically  assigned   NAVAIR  T&E    field 
activities   and  Navy  elements   of   the   DOD  Major   Range   and 
Test  Facility  Base    (MRTFB). 

(The    title    of  AIR-05  was    changed    from  Material   Acquisition  Group 

to  Systems   and  Engineering  Group   on   2  April    1979.) 

Significant   responsibilities    of  AIR-05  and   AIR-06   in 

the    control   of    field  activity   operations,    as    specified    in  NAVAIR 

Instruction   3960. 2A   are    listed   below. 

For  AIR-05: 

In  collaboration  with  AIR-06 participate  in  the  surveys 

of  candidate  T&E  field  activities to  assess  their  inaividual 

availability,  capability,  and  applicability  to  accomplish  the 
planned  T&E  programs . 

Ensure coordination  with  AIR-06  on  matters  of  T&E 

policy,  T&E  resource  requirements,  selection  of  supporting  T&E 
field  activities  and  contractors  and  funding  estimates  for 
support  of  the  DT&E  and  PAT&S  programs. 

Prepare  and  coordinate  with  AIR-06,  the  AIRTASK  and 
contract  work  assignments  for  NAVAIR  T&E  field  activities  and 
contractors . 

For  AIR-06: 

Provide  executive  management  for  all  of  the  NAVAIR  T&E 
field  activities. 

Maintain  working  Interfaces  with  all  available  T&E 
activities  including  Navy,  Army,  and  Air  Force  elements  of  the 
DOD  MRTFB,  as  well  as  the  Navy  laboratories  and  appropriate 
industrial  concerns,  in  order  to  establish  an  awareness  and 
understanding  of  the  T&E  resources  available  for  selection  to 
support  NAVAIR  T&E  programs. 

Conduct  surveys  for  the  selection  of  field  activity 
sites  for  the  conduct  of  specific  acquisition  project  T&E 
programs . 

Approve  test  site  selections . 

Render  T&E  program  support  and  assistance  to  ?MA ' s In 

the  following: 
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(d)   Expediting  T&E  reports  from  the  T&E  field 
activities. 

Provide  T&E  representation  to  the  NAVAIR  Change  Control 

Boards  to enable  planning  for  T&E  resource  support  for  the 

T&E  of  the  modification  programs  at  the  AIR-06  managed  field 
activities. 

Responsibilities  of  the  T&E  field  activities  are  as  follows: 

Provide  assistance  to  AIR-05  and  to  OPTEVFOR  and  BIS 
when  appropriate  in  structuring  the  T&E  programs  for  acquisition 
projects  and  in  planning  the  elements  thereof. 

Provide  budgetary  estimates  for  the  accomplishment  of 
T&E  workload  scheduled  for  their  respective  activities. 

Identify  T&E  resources  required  to  support  approved 
T&E  plans  (TEMP'S  and  TEP ■ s ) . 

Provide  representatives  for  site  survey  teams  when 
requested  by  NAVAIRHQ  (AIR-06). 

Perform  T&E  directed  by  NAVAIR  AIRTASK/Work  Unit 
Assignments. 

Provide  test  reports  on  a  timely  basis  to  support  the 
project  decision  making  process  at  milestone  points  identified 
In  the  approved  test  plan. 

Advise  NAVAIR  (AIR-05/06)  promptly  of  actual  or 
threatened  slippages  in  the  completion  of  work  assignments 
which  may  impact  the  scheduled  key  project  milestones  shown 
in  the  approved  test  plan  and  provide  recommendations  for 
dealing  with  the  causes  thereof. 

Advise  NAVAIR  (AIR-05/06)  promptly  when  test  results 
indicate  major  deficiencies  which  will  require  system  or 
equipment  redesign  to  enable  meeting  the  T&E  program  objectives 
stated  in  the  applicable  TEMP  or  TEP  or  which  are  likely  to 
result  in  the  breaching  of  performance,  schedule  and  funding 
thresholds  stated  in  the  applicable  DCP/NDCP. 

Responsibility  for  Introducing  the  controlling  workload 

at  Navy  MRTFB  activities  (the  T&E  field  activities  plus  portions 

of  other  Naval  Material  Command  activities)  is  assigned  to  AIR-06 

by  NAVMAT  Instruction  396O.IO.   As  such,  they  are  required  to 

perform  workload  management  and  be  responsive  to  the  technical 
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guidance  of  workload  sponsors  (primarily  AIR-05).   V/orkloati 
sponsors  are  required  to  provide  timely  notification  of  workload 
assignments  to  activity  commanders  to  ensure  proper  support. 
The  responsibilities  of  various  organizations  and  the  flow  of 
documentation  required  are  shown  in  Figures  3  and  4.  Qf] 
Figure  3  applies  to  workload  sponsored  by  NAVAIR;  whereas, 
Figure  4  applies  to  workload  sponsored  by  organizations  external 
to  NAVAIR.   It  should  be  noted  that  the  project  estimate  Is 
broken  into  direct  and  institutional  funds.   To  provide  for 
advanced  planning  the  instruction  requires  that  fiscal  planning 
information  for  the  next  five  fiscal  years  be  provided  to  AIR-06 
by  1  May  of  each  year.   The  MRTFB  activities  are  to  submit 
workload  plans  for  all  work  assigned.   As  was  mentioned  in  the 
section  on  the  Uniform  Funding  Policy,  it  is  difficult  to 
forecast  the  work  of  a  T&E  activity  even  one  year  in  advance. 
Forecasting  for  five  years  is  extremely  imprecise.   This  is 
particularly  true  because  the  forecasting  is  being  done  oy  the 
T&E  activity  itself  rather  than  the  users.   Even  knowing  what 
programs  are  scheduled  for  the  next  five  years,  it  is  often  net 
possible  to  predict  requirements  the  users  will  generate  for 
T&E  support. 

In  Figures  3  and  4,  the  portion  of  the  project  estimates 
submitted  by  the  field  activities  for  institutional  funds  ar° 
used  primarily  for  improving  facilities  and  equipment.   Unfortun- 
ately,, no  formalized  provisions  exist  for  tying  these  elements 
together  into  a  coordinated  plan  for  updating  the  facilities 
the  field  activities  separately  or  collectively.   An  Improvement 


37 


OC 

O    i/» 

2  2 

?i 

o  *• 

3    to 

UJ 

P   3 

K 

»o 

< 

2 

2   i 

w 

M 

U 

UJ 

-» 

s 

O 

UJ 

K 

V) 

E 

3 

O  u. 

m 
o 

U 

3 
bO 
«H 

fa 


•P 

£ 
O 

O 

fa 
T3 

co 
o 

r-H 
O 

3 


REVIEW 

AND 

COORDINATE 

PREPARE 
TEMP/TEP 
REQUEST  FOR 
PROJECT 
ESTIMATES 

f 

X3 

CO 

O 
i— I 

u 

o 

3 
d) 

to 

c 

co 

c 
o 

B 
o 
*-l 
-p 
co 
D 
.-l 
co 
> 
Ci3 


00 
3) 
Eh 

T3 
C 
CO 

e 

E 

o 
o 

CO 

E 
CD 
-P 
CO 

>> 

CO 


2 

O  2 
o  < 


38 


-P 
U 

c 

o 

3 
O 
i— I 

CD 

o 

«H 

id 

ti       r-H 
O        rH 

3      < 


03 
O 
rH 
J* 

tn 
O 


SI 
O 


39 


and  Modernization  (I&M)  program  has  been  established  by  AIR-06 
in  an  attempt  to  improve  this  situation.   Various  efforts  such 
as  Project  20  have  required  the  field  activities  to  project  their 
facilities  and  equipment  improvement  needs  so  that  planning  for 
these  requirements  can  be  done  in  a  centralized,  logical  manner. 
Provisions  for  incorporating  into  these  projections  requirements 
for  general  facilities  improvements  that  are  not  directly  tied 
to  projects  have  been  inadequate.   Funding  in  this  area  has  been 
very  uncertain,  probably  due  to  the  fact  that  insufficient 
justifying  information  is  available  to  secure  the  funds.   Institu- 
tional funds  for  this  purpose  are  usually  what  is  left  after  all 
other  needs  have  been  satisfied  and  are  inevitably  insufficient. 
2.   Recommendations 

To  improve  the  ability  to  plan  and  budget  the  workload 
of  the  field  activities,  closer  dialogue  between  the  activities 
and  the  users  is  recommended.   The  T&E  field  activities  should 
be  considered  to  be  and  treated  as  a  part  of  the  NAVAIR  team, 
not  subject  to  the  arm's-length  relationship  required  with 
contractors. 

To  provide  for  required  updating  of  facilities  at  the 
field  activities,  efforts  should  continue  for  a  strong  I&M 
program.   This  should  include  both  direct,  project  related,  and 
general  requirements. 

C.   THE  FIELD  ACTIVITIES 

Management  guidance  at  the  field  activities  is  primarily 
concerned  with  the  preparation  of  aggregate  information  on 
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operating  budgets  and  workload.   Another  area  of  concern  Is 
assignment  of  personnel  ceilings.   Project  work  is  nominally 
assigned  to  a  single  point  of  contact  at  the  field  activity. 
In  actuality,  however,  the  primary  project  interface  is  between 
the  operating  personnel  at  the  field  activity  and  the  "users'" 
at  Headquarters.   Because  of  this,  the  operating  people  are 
serving  two  different,  and  often  conflicting,  masters.   The 
degree  to  which  this  is  a  problem  probably  varies  between  field 
activities  for  a  variety  of  reasons,  but  particularly  the  size 
of  the  organization  and  the  degree  to  which  management  and 
operating  people  are  separated.   A  result  of  the  disconnect 
between  the  management  and  operating  personnel  is  that  while 
management  is  seeking  additional  resources  to  perform  existing 
work,  operating  personnel  are  often  seeking  additional  work  for 
existing  resources. 

D.   PRIORITY  SYSTEMS 
1.   Discussion 

Currently,  there  is  no  overall  priority  system  for 
NAVAIR.   Field  activities  establish  their  own  priorities  based 
on  informal  guidance  from  various  sources  and  their  own  percep- 
tions of  what  is  most  important.   This  has  the  advantage  of 
allowing  them  the  flexibility  to  respond  to  situations  based  on 
first-hand  knowledge  of  the  situation.   Efficient  scheduling  Is 
allowed  without  micromanagement  from  Headauarters .   Cn  the  other 
hand,  under  this  system,  priorities  tend  to  be  more  dependent 
on  the  dollar  value  of  the  test  program  than  on  its  overall 
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urgency  and  value  to  fleet  operations  and  National  defense. 
Also,  local  establishment  of  priorities  may  tend  to  ignore  Life 
Cycle  considerations  and  the  requirement  to  complete  certain 
testing  prior  to  a  scheduled  Milestone  Decision. 
2.   Recommendation 

To  date,  the  absence  of  an  overall  priority  system  has 
not  resulted  in  any  major  problems.   This  is  primarily  because 
no  major  program  has  been  delayed  significantly  due  to  lack  of 
T&E  resources.   As  the  requirements  for  T&E  programs  become  more 
formalized  and  the  length  and  complexity  increase  this  situation 
can  be  expected  to  change.   Reductions  in  resources  will  exacerbate 
the  situation  and  delays  will  be  incurred.   When  this  happens  it 
will  be  imperative  that  a  single  priority  system  is  able  to 
establish  the  order  of  precedence  of  the  various  programs.   The 
problem  has  been  recognized  and  AIR-610  has  been  working  with 
AIR-620  and  AIR-59  to  develop  a  priority  management  process. 
These  efforts  should  continue  until  a  workable  system  has  been 
devised  and  implemented. 

E.   PERSONNEL  REDUCTIONS 

Because  of  policy  changes  in  recent  years  a  continuing 
pressure  for  reduced  personnel  at  Headquarters  and  at  the  field 
activities  has  developed.   The  Office  of  Management  and  Budget 
in  Circular  A-76  stressed  the  contracting  out  of  work  to  the 
private  3ector  when  practical.   Congressional  pressures  to  reduce 
the  average  grade  level  of  civilian  employees  has  beer,  increasing 
in  recent  years.   In  response  to  these  pressures  the  level  of 
review  for  filling  of  jobs,  particularly  at  the  middle  and  upper 
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management  levels,  has  been  raised.   Quotas  on  the  filling  of 
vacancies  have  been  established  and  several  positions  have  been 
downgraded.   The  result  has  been  that  the  number  of  positions 
occupied  has  decreased  much  more  rapidly  than  ceiling  point 
reductions  would  indicate.   Furthermore,  all  of  this  is  occurring 
in  the  face  of  increasing  complexity  of  systems  being  acquired 
and  T&E  requirements  for  these  systems.   Since  little  relief  of 
the  situation  can  be  expected  in  the  near  future,  the  only  hope 
for  solution  is  to  use  the  remaining  resources  more  efficiently 
through  even  greater  centralization  of  controls  to  Headquarters. 
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IV.   DOCUMENTATION  AND  INFORMATION  SYSTEMS 

A.   AIRTASKS  AND  WORK  UNIT  ASSIGNMENTS 
1.   Discussion 

Project  work  is  currently  assigned  to  the  field  activity 
by  Airtask.   The  policy  and  procedures  for  AIRTASKS  are  established 
by  NAVAIRINST  3900. 8A.  £l|]  The  AIRTASK,  generally  no  more  than 
two  pages  in  length,  represents  an  outline  of  the  work  to  be 
performed.   It  includes  background  information,  detailed  require- 
ments, source  and  disposition  of  project  equipment  and  aircraft, 
reports  required,  and  a  funding  estimate.   The  background 
information  discusses  events  leading  to  the  project  and  estab 
the  need.   The  detailed  requirements  provide  a  list  of  the 
specific  functions  to  be  performed.   These  are  usually  very  broad 
and  provide  maximum  flexibility  for  Interpretation  as  the  project 
progresses  and  the  perception  of  the  details  of  the  reauirements 
change.   The  source  and  disposition  of  project  equipment  and 
aircraft  usually  only  states  that  existing  assets  should  be  used, 
that  they  will  be  provided,  or  that  requirements  should  be 
coordinated  with  the  NAVAIRSYSCOM  Aircraft  Custodian.   The  reports 
required  may  vary  greatly  but  usually  include,  as  a  minimum,  a 
final  report  and  recently,  a  detailed  project  plan  to  be  submitted 
early  in  the  prosecution  of  the  project.   The  funding  estimate 
states  the  amount  of  money  to  be  provided  for  the  project  by 
separate  correspondence,  usually  a  work  request,  from  Comptroller 
to  Comptroller. 
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Work  Unit  Assignments  are  subordinate  elements  of  an 
AIRTASK  for  the  performance  of  a  specific  task  within  the  scope 
of  the  previously  assigned  AIRTASK.   The  same  form  (NAVAIR  3930/1) 
is  used  for  both  AIRTASKS  and  Work  Unit  Assignments. 

To  help  ensure  efficient  utilization  of  resources  NAVAIRINST 
3900.8A  requires  that  all  assignments  Involving  T&E  be  coordinated 
with  AIR-620,  the  T&E  Projects  Division  of  AIR-06.   This  is  an 
important  step  in  centralizing  the  control  of  field  activity 
operations  and  promoting  optimum  use  of  their  resources.   On  the 
other  hand,  it  reduces  the  authority  of  the  technical  managers 
to  deal  directly  with  the  field  activities  for  work  to  be 
performed.   For  this  reason  the  policy  is  followed  reluctantly, 
if  at  all. 

The  AIRTASK/Work  Unit  Assignment  includes  a  section  for 
detailed  requirements.   Quite  often  this  section  receives 
inadequate  attention  so  that  there  is  not  a  clear  understanding 
between  the  originator  and  field  activity  personnel  of  what  is 
required.   This  may  be  alleviated  somewhat  by  referencing  a  work 
unit  plan  which  has  been  generated  by  the  field  activity.   This 
is  satisfactory  only  if  the  person  directing  the  work  has  read 
and  understood  the  work  unit  plan  and  concurs  in  what  it 
proposes.   This  is  not  always  the  case.   Another  method  which 
may  be  used  to  promote  understanding  of  the  requirements  Is  to 
require  the  field  activity  to  submit  a  detailed  program  plan. 
Unfortunately,  the  instruction  does  not  stress  that  this  should 
be  an  iterative  process,  continuing  until  an  agreement  has  been 
reached.   All  of  this  may  be  time  consuming  and  may  be  intolerable 
on  urgent  projects. 
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2.   Recommenda  tions 

To  avoid  misunderstanding  and  to  help  preclude  completion 
of  the  test  program  without  satisfying  the  objectives,  discussions 
should  be  held  and  proposed  AIRTASKS/VORK  UNITS  should  be  reviewed 
prior  to  formal  promulgation  of  the  AIRTASK/WCRK  UNIT.   This 
procedure  should  also  be  followed  for  detailed  project  plans 
when  they  are  required.   The  discussions  should  include  the  user, 
the  field  activity,  and  AIR-620  representatives. 

B.   TEST  PLANS 

1.  Discussion 

Currently  there  is  no  standard  method  or  format  for  test 
plans  common  to  all  field  activities,  nor  is  there  a  requirement 
to  submit  the  test  plan  for  review.   This  allows  the  field 
activities  maximum  latitude  to  establish  procedures.   This  is 
particularly  important  in  the  area  of  flight  safety  where  each 
activity  assumes  accident  reporting  responsibility  and,  therefore, 
must  have  the  authority  to  establish  their  own  safety  procedures. 

2.  Recommenda  tions 

In  spite  of  the  need  for  maximum  flexibility  at  the  field 
activities,  there  is  also  a  need  for  closer  control  by  project 
personnel  at  headquarters.   Field  activity  personnel  are  not 
always  aware  of  important  considerations  which  must  be  emphasized 
in  the  test  program.   To  help  ease  this  problem  a  base  line  format 
for  test  plans  should  be  established  which  the  field  activities 
could  modify  to  suit  their  own  individual  requirements.   Test 
plans  should  be  required  by  the  AIRTASK/Work  Unit  Assignment  to 
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be  submitted  for  review.   However,  this  provision  must  be  used 
carefully  to  avoid  loss  of  field  activity  flexibility  and 
independence  and  to  maintain  an  objective  test  program. 

C.   REPORTING  THE  RESULTS  OF  T&E 
1.   Discussion 

For  T&E  to  be  of  any  value  the  results  must  be  reported 
to  the  decision  makers  in  a  timely  manner.   Also,  they  must  be 
reported  in  such  a  way  that  they  may  be  easily  and  rapidly 
understood  by  someone  who  does  not  have  a  great  deal  of  time  to 
read  lengthy  reports.   In  the  case  of  discrepant  performance, 
the  sooner  the  discrepancy  is  reported  and  action  is  taken  to 
correct  it,  the  less  costly  the  correction  is  likely  to  be.   The 
further  the  development  or  production  process  proceeds  before  a 
discrepancy  is  discovered,  the  more  that  must  be  corrected  once 
it  is  discovered. 

No  standard  procedures  have  been  established  for 
reporting  the  results  of  T&E  by  the  various  field  activities. 
Reports  have  been  late  and  of  little  value  because  of  the 
lateness.   Information  that  has  been  used  has  been  passed  through 
briefings,  letters,  and  messages.   The  report  has  only  served  as 
a  record  of  what  has  been  accomplished  and  is  used  very  little 
as  the  basis  for  decisions.   Under  this  system,  only  major 
discrepancies  are  reported  in  a  timely  manner.   Minor  problems, 
which  taken  together  may  have  major  Impact,  are  generally  not 
reported  until  the  test  program  is  complete  when,  in  some  cases, 
it  may  be  too  late  to  correct  them. 
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Procedures  have  not  been  developed  at  Headquarters  to 
process  incoming  reports  and  make  them  readily  available  to 
anyone  who  needs  them.   They  go  to  the  code  sponsoring  the  work 
and  other  codes  designated  by  the  field  activity  originating  the 
report,  but  not  necessarily  to  all  offices  requiring  knowledge 
of  the  test  results.   No  system  exists  that  can  be  accessed  by 
anyone  needing  the  information  to  tell  if  a  report  has  been 
submitted  on  a  project,  and,  if  so,  how  to  locate  the  report. 

Field  activity  emphasis  has  been  on  current  testing  with 
reports  being  written  and  reviewed  as  the  schedule  allows.   This 
is  as  much  due  to  the  reluctance  of  personnel  to  do  the  writing 
as  it  is  due  to  any  established  policies.   To  overcome  this 
reluctance,  methods  could  be  devised  to  make  -the  writing  simpler 
and  shorter.   The  use  of  deficiency  reports  is  a  good  example  of 
this.   The  deficiency  reports  are  brief  enough  so  that  they  can 
be  easily  generated  with  minimum  detail  and  be  modified  as  more 
is  learned  by  submitting  additional  reports.   They  are  submitted 
as  problems  are  noted,  not  at  the  end  of  the  test  program.   The 
use  of  deficiency  reports,  however,  greatly  increases  the  number 
of  reports  which  must  be  tracked,  including  the  actions  taken 
as  a  result  of  the  reports.   This  increases  the  requirement  for 
a  system  to  record  all  reports  so  that  they  may  be  accessed 
rapidly. 

2.   Recommenda  tions 

To  improve  the  usefulness  of  T&E  reports,  standard 
procedures  should  be  established  for  their  submission.   This 
should  include  a  standard  report  format  and  should  be  based  on 
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a  system  similar  to  that  currently  used  for  deficiency  reports. 
Reporting  requirements  must  be  kept  brief  to  allow  timeliness. 
As  a  part  of  an  overall  T&E  information  system,  procedures  should 
be  established  to  track  all  reports  for  easy  access  and  to  track 
actions  taken  as  a  result  of  the  reports. 

D.   WORK  UNIT  SUMMARIES  AND  FIELD  ACTIVITY  PLANS 

Navy  MRTFB  activities  are  required  by  Reference  11  to  submit 
Work  Unit  Summaries  for  all  current  and  expected  projects  to 
AIR-06  for  use  in  planning  and  preparing  the  MRTFB  budget  and  in 
the  management  of  direct  workload  at  the  MRTFB  activities. 
Included  are  appropriation  and  project  information  and  estimates 
of  the  manpower  and  funding  requirements  of  the  project.   Also, 
included  is  Information  on  the  backgrouna,  objectives  ana  progress 
of  the  project.   Reference  11  also  directs  that  workload  sponsors 
provide  wriuten  fiscal  guidance  to  AIR-06  and  the  MRTFB  activities 
If  the  system  works  properly,  the  Work  Unit  Summaries  should 
reflect  this  fiscal  guidance.   It  has  been  the  author's  experience 
that  this  is  rarely  the  case;  however,  the  instruction  is 
relatively  new  and  the  situation  may  improve  with  time. 

Detailed  back-up  information  for  the  manpower  and  cost 
estimate  of  the  Work  Unit  Summary  is  provided  by  the  Field 
Activity  Plan.   The  manpower  estimates  are  broken  down  by 
functional  area  and  by  type  of  personnel  (military  or  civilian.). 
The  cost  estimates  are  broken  into  manpower,  material,  flight 
hours,  travel,  contract,  and  funding  to  other  activities. 
Obviously,  this  information  must  be  balanced  against  the  fiscal 
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guidance  from  the  workload  sponsors  and,  theoretically,  the 
fiscal  guidance  should  reflect  the  cost  estimate  if  there  is  to 
be  any  agreement  on  what  is  to  be  done  for  how  much.   Again, 
there  is  little  indication  that  this  is  the  case,  but  hopefully 
the  situation  will  improve. 

E.   COMPREHENSIVE  INFORMATION  SYSTEM 
1.   Discussion 

Many  of  the  problems  discussed  to  this  point  result  from 
poor  information  flow.   Much  of  this  could  be  improved  with 
revision  and  integration  of  the  information  systems  used  for  T&E 
field  activity  management  into  a  unified  comprehensive  information 
system. 

It  is,  therefore,  worthwhile  to  examine  the  environment 
in  which  such  an  information  system  must  operate,  existing  systems 
and  the  feasibility  of  establishing  an  information  system  which 
will  allow  efficient  management  of  resources  at  the  Headquarters 
and  field  activity  levels. 

Currently  various  systems  have  been  established  at  the 
field  activities,  primarily  to  manage  their  own  internal 
resources  and  to  provide  accounting  for  the  funding  allocated 
for  various  projects.   In  terms  of  the  overall  NAVAIR  organization 
these  are  primarily  operations  control  systems  with  some  lower 
level  management  control  applications.   To  better  control  the 
test  and  evaluation  process  the  TEMP  is  required  for  each  major 
acquisition  project.   The  TEMP  outlines  the  testing  required, 
the  schedule,  and  required  resources.   Field  Activity  Plans  and 
Work  Unit  Summaries  outline  the  financial,  manpower,  and  squlpne 
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resource  requirements  and  the  approach  to  be  taken.   None  of 
these  documents,  however,  provides  a  complete  picture  of  the 
capabilities,  resources  available,  resources  required,  and 
workload  of  each  of  the  field  activities  individually  or  in 
aggregate.   The  feasibility  of  establishing  such  a  system  should 
be  investigated. 

The  groundwork  has  now  been  established  for  the  examina- 
tion of  the  environment  of  such  a  system  in  the  NAVAIR  organi- 
zation.  Size  of  the  organization  does  net  appear  as  a  deterrent 
to  establishment  of  an  information  system.   In  fact,  just  the 
opposite  is  true.   The  large  and  diverse  nature  would  seem  to 
dictate  that  an  information  system  be  established  if  adequate 
planning  of  resource  allocation  and  utilization  is  to  be 
accomplished. 

The  organizational  structure  of  NAVAIR  and  its  field 
activities  may  represent  one  of  the  most  formidable  barriers  to 
the  establishment  of  an  effective  Information  system.   The 
organization  Is  extremely  decentralized.   Each  of  the  field 
activities  has  its  own  command  structure  with  its  own  motives 
which  in  actuality  may  or  may  not  align  with  the  overall  NAVAIR 
goals.   Further  the  field  activities  report  to  no  single  authority 
within  headquarters.   Nor  do  headquarters  personnel  deal  with 
single  points  of  contact  at  the  field  activities.   Different 
headquarters  offices  are  involved  depending  on  whether  the 
matters  in  question  are  associated  with  the  conduct  of  individual 
projects  or  overall  administrative  control  and  support.   The 
headquarters  personnel  associated  with  specific  project  areas 
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generally  deal  directly  with  field  activity  personnel  who  will 
be  working  on  their  projects.   Because  of  these  factors  an 
extreme  amount  of  coordination  and  cooperation  would  be  required 
if  a  usable  information  system  is  to  be  established. 

Several  factors  effect  the  time  frame  within  which  an 
information  system  would  need  to  be  constructed.   NAVAIR,  in  its 
role  as  developing  agency,  must  deal  with  continuous  change. 
Technology  is  changing  rapidly  and  continuously,  but  also  the 
regulations  regarding  systems  acquisition  and  thus  the  methods 
that  are  required  to  be  used  are  al30  continually  changing.   This 
often  dictates  that  the  interface  with  the  field  activities  change 
from  time  to  time.   These  changes,  however,  are  usually  relatively 
minor,  and  the  relationship  between  NAVAIR  and  its  field  activities 
has  not  changed  substantially  in  several  years.   Field  activity 
control  can  probably  be  made  more  efficient  and  effective  through 
the  use  of  an  information  system;  however,  time  constraints  do 
not  appear  to  be  such  that  pressure  exists  for  a  development 
project  that  must  be  rushed  to  the  exclusion  of  adequate  planning. 

The  organization  in  which  the  information  system  must 
operate  is  somewhat  difficult  to  define.   There  are  organizational 
segments  which  clearly  have  a  large  role  as  the  prime  contributors 
and  users.   However,  there  are  other  segments  of  the  organization 
that  will  impact  the  operation  whose  participation  is  not  clearly 
defined.   The  prime  headquarters  users  of  the  system  would  be 
AIR-06;  however,  since  a  large  portion  of  the  interface  between 
NAVAIR  and  the  field  activities  includes  the  assignment  of  work 
by  other  parts  of  the  organization,  their  inputs  to  the  system 


are  also  required.   The  relationships  between  these  organiza- 
tional components  is  undergoing  changes  which  will  take  some 
time  to  settle  into  focus.   Until  this  occurs  the  organization 
cannot  be  considered  mature  and  this  must  be  recognized  as  a 
possible  stumbling  block  to  the  establishment  of  a  successful 
information  system. 

NAVAIR  Headquarters  does  not  have  in-house  hardware  or 
personnel  to  be  utilized  for  the  development  and  operation  of 
the  information  system.   Hardware  and  many  of  the  personnel 
assets  required  exist  at  the  field  activities.   Hardware  could 
probably  be  made  available  on  a  time-sharing  basis;  however, 
whether  or  not  personnel  could  be  made  available  for  the  project 
would  require  some  study.   Also,  it  may  be  undesirable  to  have 
any  one  of  the  field  activities  established  as  the  central 
location  for  information  system  data  storage  and  analysis.   A 
possible  alternative  is  to  contract  the  development  and  operation 
of  the  information  system  to  an  outside  consulting  organization. 
This  is  attractive  since  contractual  mechanisms  could  be  used  to 
help  control  cost  and  schedule.   The  real  problem  arises  in 
finding  sufficient  funding  to  adequately  support  the  effort. 
Since  the  field  activity  control  task  is  currently  being  performed 
without  major  difficulties  being  apparent  at  levels  controlling 
the  funds,  a  major  effort  will  be  required  to  sell  the  benefits 
of  the  system.   One  possibility  to  accomplish  this  might  be  tc 
cite  examples  of  past  inefficiencies  and  their  associated  costs 
which  could  have  been  avoided  with  the  proposed  information 
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system  in  operation.   A  side  benefit  of  this  would  be  to  help 
define  the  areas  toward  which  the  informa  tior.  system  should  be 
directed. 

Attitudes  about  information  systems  are,  unfortunately, 
generally  hostile.   Too  often  the  same  data  have  been  required 
to  be  submitted  several  times  in  different  formats  to  satisfy 
different  information  systems  only  to  have  the  results  of 
considerable  effort  go  unused  for  any  significant  decisions 
regarding  the  operation  of  the  organization.   Expectations  for 
an  information  system  would  probably  be  low;  therefore,  any 
demonstration  of  successful  operation  would  probably  come  as  a 
welcome  surprise.   One  of  the  biggest  problems  is  convincing 
the  operations  level  people  who  make  most  of  the  inputs  to  the 
system  that  it  is  worth  their  while  to  spend  the  time  and  effort 
to  make  the  inputs  accurate.   This  could  be  significantly  aided 
if  the  same  information  is  required  cnly  once  and  if  pa3t 
information  submitted  may  be  easily  reviewed  and  updated  without 
complete  resubmission.   Also,  it  is  necessary  that  they  be 
convinced  that  the  results  of  their  efforts  serve  some  useful 
purpose.   For  these  reasons  it  is  important  that  the  managers 
who  are  potential  users  of  the  system  be  convinced  that  it  la 
worth  their  effort  to  stay  involved  in  the  development  of  the 
system  beyond  the  establishment  of  the  initial  requirements. 

Assuming  that  in  spite  of  the  draw  backs  cited  in  the 
preceding  paragraphs,  it  is  decided  that  an  information  system 
for  the  controlling  of  field  activity  operations  is  worth  the 
effort,  it  is  possible  to  look  at  what  might  be  some  of  the 
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elements  of  the  Information  system  target.   At  least,  the 
definition  and  some  of  the  planning  stages  can  be  examined. 

Various  systems  have  already  been  established  at  the 
field  activity  or  operational  level  which  may  be  modified  or 
used  intact  as  Inputs  to  the  Information  system.   Some  direction 
from  headquarters  as  to  the  type  of  information  required  in  the 
system  is  also  required.   For  these  reasons  the  parallel  approach 
of  Reference  14,  where  operational  systems  and  planning  and 
control  systems  are  developed  simultaneously,  seems  to  be  appro- 
priate for  this  information  system  development.   Top  down  is 
ruled  out  because  it  would  probably  require  more  top  management 
support  than  is  available.   This  would  be  particularly  true  when 
attempting  to  integrate  information  from  the  various  parts  of 
the  matrix  organization.   On  the  other  hand,  bottom  up  would  be 
easier  to  accomplish,  but  due  to  differences  between  the  field 
activities  it  is  doubtful  that  a  system  with  sufficient  integration 
to  be  useful  could  be  attained  this  way. 

There  is  little  requirement  for  the  field  activities  to 
exchange  information  between  themselves;  therefore,  there  is  no 
requirement  for  a  high  degree  of  integration  at  this  point.   They 
should,  however,  be  able  to  pass  the  same  information  in  the 
same  formats  to  headquarters  for  further  aggregation  and  analysis. 
Although  this  is  desirable  it  does  not  mean  that  nothing  useful 
can  be  gained  treating  the  information  received  from  each  of  the 
field  activities  by  itself.   For  example,  aircraft  requirements 
for  current  and  future  projects  can  be  analyzed  for  each  field 
activity  individually.   On  the  other  hand,  requirements  fcr 
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funding  for  new  instrumentation  and  facilities  mu3t  be  compared 
for  all  activities  to  establish  priorities  for  allocation  of 
extremely  scarce  resources. 

Because  of  the  psychological  climate  in  which  the  system 
must  operate,  innovation  and  the  attendant  risk  should  be  kept 
to  a  minimum.   It  is  important  to  achieve  early,  if  modest, 
successes  to  maintain  and  improve  support  for  the  3ystem.   One 
area  where  some  innovation  may  be  allowed  is  data  base  management. 
This  technique  offers  considerable  appeal  in  terras  of  establishing 
the  system  such  that  a  given  piece  of  data  information  needs  to 
be  entered  into  the  system  only  once,  which  will  make  the  system 
attractive  to  those  who  must  feed  it. 
2.   Recommenda  tion 

The  priorities,  specific  functions,  and  goals  of  the 
projected  information  system  require  considerable  in-depth 
investigation  beyond  the  scope  of  this  thesis.   These  could  be 
developed  using  the  preceding  discussion  and  discussions  with 
personnel  who  would  be  involved  in  the  operation  and  use  of  the 
system.   A  study  should  be  established  involving,  as  a  minimum, 
representatives  of  AIR-05,  AIR-06,  and  the  T&E  field  activities. 
High  level  attention  must  be  maintained  to  ensure  proper  support 
for  the  system  instituted  as  the  result  of  this  study.   In  the 
establishment  of  the  information  system  a  consultant  familiar 
with  the  establishment  of  information  systems  should  be  hired  to 
prevent  many  of  the  pitfalls  which  might  not  be  seen  by  someone 
unfamiliar  with  the  process  of  establishing  an  information  system. 


56 


Emphasis  in  the  development  should  be  placed  on  ease  of  submission 
and  updating  of  information  and  information  requirements  should 
be  limited  to  those  for  which  a  valid  use  has  been  identified. 
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V.   PROBLEM  AREAS  AND  OPPORTUNITIES 

A.   ACTIVITY  MISSION  STATEMENT  OVERLAP 
1.   Discussion 

There  are  four  T&E  field  activities,  listed  In  Table  1, 
under  the  administrative  control  of  AIR-06.   In  addition  signi- 
ficant NAVAIR  T&E  functions  are  performed  at  the  Naval  Weapons 
Center  and  the  Atlantic  Undersea  Test  and  Evaluation  Center, 
although  they  are  not  NAVAIR  field  activities.   The  missions  of 
the  field  activities  are  stated  In  NAVAIR  Instruction  5451.82. 
Some  areas  are  clearly  within  the  mission  of  only  one  of  the 
field  activities.   In  other  areas,  however,  there  is  considerable 
overlap  in  the  stated  missions.   As  an  example,  the  mission 
statements  of  the  two  largest  field  activities,  the  Naval  Air 
Test  Center  (NATC)  and  the  Pacific  Missile  Test  Center  (PMTC) 
are  quoted  below.   For  NATC  the  mission  is: 

"To  conduct  tests  and  evaluation  of  aircraft  weapons 
systems  and  their  components."  £L5j 

For  PMTC  the  mission  is: 

"To  perform  development  test  and  evaluation,  develop- 
ment support,  follow-on  engineering,  logistics  and  training 
support,  for  naval  weapons,  weapons  systems  and  related 
devices;  provide  major  range,  technical  and  base  support 
for  Fleet  users  and  other  DOD  and  government  agencies."  [15] 

This  is  only  one  example.   There  are  overlaps  with  other 

activities  also.   NAVAIRINST  5451.82  also  states  the  specific 

functions  to  be  performed  by  the  field  activities  in  the  accomplish' 

ment  of  their  missions.   Here  there  is  further  overlap. 
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Table  1 

NAVAL  AIR  SYSTEMS  COMMAND 
TEST  AND  EVALUATION  FIELD  ACTIVITIES 

Naval  Weapons  Evaluation  Facility,  Albuquerque,  N.M. 
Naval  Air  Propulsion  Center,  Trenton,  N.J. 
Naval  Air  Test  Center,  Patuxent  River,  MD. 
Pacific  Missile  Test  Center,  Point  Mugu,  C. 

The  result  of  this  overlap  is  a  certain  amount  of 
wasteful  duplication  of  facilities  and  capabilities  and  the 
competition  between  field  activities  cited  by  the  Civilian 
Management  Board/Middle  Management  Council  [TJ  and  discussed  in 
the  Introduction.   Certainly,  some  of  the  duplication  is 
necessary.   All  test  ranges  must  have  position  measurement, 
communications,  and  data  gathering  and  processing  systems.   Also, 
a  certain  amount  of  competition  is  inevitable  and,  in  fact, 
beneficial  because  It  helps  to  keep  the  field  activities  effi- 
cient and  responsive  to  the  needs  of  Headquarters.   The  key  to 
whether  the  duplication  is  wasteful  or  not  is  In  the  amount  of 
marketing  done  by  the  field  activities  for  work  and  the  degree 
to  which  the  capacity  of  the  field  activities,  as  a  group,  is 
utilized.   The  marketing  that  accompanies  the  competition  between 
the  activities  often  results  In  waste  because  work  is  performed 
that  is  not  really  necessary.   This  would  seem  to  indicate  that 
capacity  is  not  being  utilized  efficiently  and  some  wasteful 
duplication  does  exist.   This  is  paradoxical  because  in  many 
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cases  the  field  activities  claim  to  be  overworked,  underfunded, 
and  understaf fad,  yet  the  inertia  of  the  system  causes  them  tc 
continue  to  seek  new  work.   In  fact,  at  different  levels  in  the 
organization  there  exist  different  perceptions  of  what  the  work 
level  really  is.   One  level  of  management  may  be  actively  pursuing 
new  projects  while  another  level  is  trying  to  discourage  them. 
2.   Recommendations 

It  would  appear  that  one  way  to  decrease  the  wasteful 
results  of  competition  is  greater  centralization  of  T&E  project 
control  in  AIR-06  where  a  greater  knowledge  of  the  overall 
capabilities  and  workloads  of  the  field  activities  exists. 
However ,  it  Is  unlikely  that  this  would  work  for  two  reasons. 
First,  there  are  insufficient  personnel  in  AIR-06  to  assume  the 
added  responsibility  and  it  is  unlikely  that  additional  personnel 
are  available.   Second,  this  is  a  reduction  in  the  flexibility 
and  authority  of  AIR-05  and  would  meet  strong  resistance  from 
that  quarter.   The  best  solution  is  probably  merely  to  encourage 
greater  awareness  of  the  problem  and  greater  cooperation  between 
AIR-05  and  AIR-06  through  the  policies  established  in  NAVKATINST 
3960.10  and  NAVAIRINST  3960. 2A .   Proposals  have  been  made  for 
the  establishment  of  a  facilities  ard  capabilities  handbook  which 
could  be  used  to  increase  the  awareness  of  all  T&E  facilities 
users,  and  particularly  AIR-05,  of  the  areas  of  expertise  of  the 
various  T&E  field  activities.   This  would  improve  the  situation 
and  such  a  handbook  should  be  published. 
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B.   PLANNING  AND  BUDGETING  FACILITIES  IMPROVEMENTS 
1.   Discussion 

The  Uniform  Funding  Policy  and  NAVMATIN3T  3960.10 
require  that  facilities  improvements  that  support  a  specific 
project  be  paid  for  by  funds  from  that  project  and  that  general 
improvements  in  capabilities  be  paid  for  by  the  MRTFB  budget. 
In  practice  it  is  often  difficult  to  place  a  given  improvement 
in  one  category  or  the  other.   The  result  is  that  the  "user" 
refuses  to  pay  to  improve  the  overall  capability  of  the  activity 
and  AIR-06  refuses  to  pay  for  improvements  to  support  a  project 
and  needed  improvements  are  not  accomplished.   Further,  there 
is  no  provision  for  facilities  improvements  which  cannot  be 
justified  by  an  immediate  application.   This  precludes  long 
range  planning  and  efficient  modernization  programs  because 
accurate  predictions  of  future  projects  are  not  available. 

The  other  difficulty  with  funding  facilities  improvements 
from  the  MRTFB  budget,  mentioned  earlier,  is  the  low  priority 
of  facilities  improvements  when  compared  with  other  demands  for 
MRTFB  funds  such  as  salaries  and  other  operating  expenses  which 
are  relatively  fixed  and  must  be  paid.   When  user  funding  falls 
short  of  budget  estimates  it  is  the  facility  improvements  which 
inevitably  suffer.   Even  when  user  funding  is  adequate,  MRTFB 
funding  must  be  held  in  escrow  against  possible  shortfalls.   This 
results  in  hurried  spending  at  year  end  instead  of  a  balanced 
program  throughout  the  year. 
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2.   Recommenda  ticns 

To  alleviate  these  problems  and  allow  facilities 
improvements  to  be  based  on  a  coordinated  long  range  plan,  the 
funding  of  facilities  improvements  should  be  exempted  from  the 
Uniform  Funding  Policy  and  funded  entirely  from  MRTFB  funds 
except  when  large  dollar  value  improvements  are  being  considered 
for  a  specific  project.   In  this  case,  "user"  funding  might  be 
used  to  supplement  MRTFB  funding.   The  negotiation  for  "user" 
funding  should  be  between  AIR-06  and  the  program  manager.   The 
field  activities  should  not  be  involved.   Further,  the  IScM 
portion  of  the  MRTFB  budget,  which  funds  the  facilities  improve- 
ments, should  become  a  separate  allocation  not  subject  to  other 
uses.   This  would  result  in  a  reduction  in  the  flexibility  of 
the  field  activities  to  utilize  the  MRTFB  funds  as  required  but 
it  would  help  to  ensure  that  needed  improvements  would  be 
provided. 
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VI.   SUMMARY  OF  RECOMMENDATIONS 

The  recommendations  which  have  been  made  generally  fall  into 
one  of  three  categories;  changes  in  operating  procedure,  changes 
in  budgetary  procedure,  and  emphasis  on  implementing  existing 
policies.   These  recommendations  are  summarized  In  the  sections 
that  follow. 

A.   OPERATING  PROCEDURE  RECOMMENDATIONS 

Dialogue  between  users  and  the  T&E  field  activities  should 
be  increased.  Field  activities  should  be  considered  a  part  of 
the  NAVAIR  team  rather  than  as  contractors. 

A  single  Integrated  priority  system  for  T&E  field  activity 
support  should  be  established. 

Detailed  discussions  should  be  held  to  review  AIRTASKS/WCRK 
UNITS  and  detailed  program  plans  prior  to  their  promulgation. 

A  standard  format  should  be  developed  for  test  plans  and 
they  should  be  submitted  for  review  by  the  work  sponsor. 

Standard  procedures  should  be  developed  for  reporting  the 
results  of  T&E.   These  should  be  based  on  simplified  requirements 
which  minimize  the  effort  involved  in  preparing  the  report, 
which  would  improve  timeliness.   A  report  tracking  system  should 
be  established  at  Headquarters  to  provide  easy  access  to  the 
reports  and  status  of  required  actions. 

A  study  effort  should  be  instituted  which  would  result  in 
the  establishment  of  a  NAVAIR  T&E  information  system. 
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A    facilities   and   capabilities   handbook   should   be   prepared 
to   improve   awareness    of  users   of  areas    of  expertise    of   the 
various   T&E    field  activities. 

B.  BUDGETARY  PROCEDURE  RECOMMENDATIONS 

The  requirement  to  obligate  all  RDT&EN  funds  within  the 
first  year  of  their  availability  should  be  reexamined  to  determine 
if  it  causes  more  harm  than  good. 

Funding  of  facilities  requirements,  except  in  rare  cases, 
should  be  paid  for  entirely  from  MRTFB  funding. 

The  I&M  portion  of  the  MRTFB  budget  should  be  a  separate 
allocation,  not  subject  to  other  uses. 

C.  POLICIES  WHICH  REQUIRE  FURTHER  EMPHASIS 

Users  of  T&E  field  activities  should  submit  written  fiscal 
planning  information  to  AIR-6103  per  NAVMATINST  3960.IO. 

Efforts  for  a  strong  I&M  program  should  continue. 

To  minimize  wasteful  duplication  of  facilities  and  capabil- 
ities and  competition  between  T&E  field  activities,  policies 
of  NAVMATINST  396O.IO  and  NAVAIRINST  3960.2A  must  be  followed 
more  closely. 
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APPENDIX  A 

ELEMENTS  OF  THE  MAJOR  RANGE  AND  TEST  FACILITY  BASE   £s] 
NAVY 

Pacific  Missile  Test  Center 

Atlantic  Undersea  Test  and  Evaluation  Center 

Naval  Air  Test  Center 

Naval  Air  Propulsion  Center 

Naval  Weapons  Center 
(T&E  Portion  Only) 

Atlantic  Fleet  Weapons  Training  Facility 

AIR  FORCE 

Eastern  Test  Range 

Space  &  Missile   Test  Center 

Satellite  Control  Facility 

Arnold  Engineering  Development  Center 

4950th  Test  Wing 

Tactical  Fighter  Weapons  Center 

Flight  Test  Center 

Armament  Development  and  Test  Center 

Air  Defense  Weapons  Center 

ARMY 

Cold  Regions  Test  Center 
Tropic  Test  Center 
Yuma  Proving  Ground 
Jefferson  Proving  Ground 
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White  Sands  Missile  Range 
Kewajalein  Missile  Range 
Dugway  Proving  Ground 
Aberdeen  Proving  Ground 

NOTE:   The  Naval  Weapons  Evaluation  Facility  is  a  NAVAIR  T&E 
field  activity  not  included  in  the  MRTFE. 
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APPENDIX  B 
DEFINITIONS  OF  ACQUISITION  CATEGORIES   Q.2] 

ACAT  I:   Projects  designated  by  the  Secretary  of  Defense 
(SECDEF)  or  Deputy  Secretary  of  Defense  (DEP3ECDEF)  having  a 
nominal  dollar  value  in  excess  of  $75M  RDT&E  or  $200M  Production 
and  such  other  lesser  programs  as  may  be  so  designated  by  SECDEF 
or  DEPSECDEF. 

ACAT  II:   Projects  designated  by  a  Defense  Systems  Acquisition 
Review  Council  (DSARC)  principal,  SECNAV  or  CNO  having  a  nominal 
dollar  value  in  excess  of  $20M  RDT&E  or  $50M  Production  and  such 
other  lesser  programs  as  may  be  recommended  by  the  Chief  of 
Naval  Material  (CHNAVMAT),  CP-OSO,  OP-O98  or  the  program  sponsors 
(DCNO/DMSO). 

ACAT  III:   Projects  designated  by  the  OPNAV  sponsor,  having 
a  nominal  dollar  value  in  excess  of  $5M  RDT&E  or  $20M  Production 
and  such  other  lesser  programs  recommended  by  CHNAVMAT,  OP-090, 
OP-O98,  or  the  DA.   Programs  below  the  ACAT  III  threshold  will 
normally  be  designated  ACAT  III  if  they: 

(a)  Directly  affect  the  military  characteristics  of 
ships,  aircraft  or  other  combat  units; 

(b)  Require  Operational  Test  and  Evaluation  (CT&S)  to 
support  key  program  decisions;  or 

(c)  Require  Fleet  RDT&E  support. 

ACAT  IV:   Projects  not  designated  as  ACAT  III  or  higher 
which  meet  any  one  of  the  following  criteria: 

(a)  Have  a  unit  cost  of  more  than  $10,000,  net 
including  management  and  support  costs; 

(b)  Have  a  total  project  cost  of  $1M  (R&D  plus 
production)  excluding  shore  based  training  devices  and  shore 
support  equipment  not  used  in  direct  support  of  deployed 
systems; 

(c)  The  hardware  required  formal  aooroval  for  service 
use  (ASU). 
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APPENDIX  C 
DIRECTIVES  AND  INSTRUCTIONS 


DOD  Directive  3200.11,  Use,  Management  and  Operation  of 
Department  of  Defense  Major  Ranges  and  Test  Facilities, 
June  18,  1974. 

DOD  Directive  5000.1,  Major  System  Acquisition,  January  13,  1977. 

DCD  Directive  5000.2,  Major  System  Acquisition  Process,  January 
18,  1977. 

DOD  Directive  5000.3.  Test  and  Evaluation,  January  1,  1973. 

SEC  NAVINST  5000.1,  System  Acquisition  in  the  Department  of 
the  Navy.  — 

OPNAVINST  3900. 25A. 

OPNAVINST  3960.IO,  Test  and  Evaluation. 

OPNAVINST  5440. 47D,  Mission  and  Functions  of  Operational  Test 
and  Evaluation  Force  (OPTEVFOR),  6  March  1973. 

NAVMATINST  3910. 15A. 

NAVMAT  INST  396O.6A,  Test  and  Evaluation. 

NAVMATINST  3960.8,  Land-Eased  Test  Site  (L3TS)  Selection  Policy, 
1  September  1976. 

NAVMATINST  396O.IO,  Policy  and  Procedures  for  Workload  Manage- 
ment at  the  Navy  Major  Range  and  Test  Facility  Base  (MRTFB) 
Activities,  3  February  1978. 

NAVMATINST  5450.31,  Consolidation  of  Navy  Ranges,  Test 
Activities,  and  Target  Test  Resources,  29  March  1976. 

NAVAIRINST  39OO.8A,  AIRTASK  and  Work  Unit  Assignments  to  Field 
Activities,  22  March  1977. 

NAVAIRINST  396O.2A,  Test  and  Evaluation,  11  August  1978. 

NAVAIRINST  5451.82,  Naval  Air  Systems  Command  (NAVAIR)  Test  and 
Evaluation  Activities;  Mission  and  Functions  of,  lo  December  19"? 
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